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Key  findings 

Psychological  health  quality  measurement  is  common  in 
health  plans. 

Health  plans  primarily  focus  on  collecting  and  reporting 
measures  of  the  process  of  care. 

Health  plans  use  quality  measures  to  improve  the  qual¬ 
ity  of  psychological  health  care. 

Measuring  the  quality  of  psychological  health  care  is 
challenging. 

Measuring  the  value  of  psychological  health  care  is 
more  difficult  and  less  common. 


SUMMA R Y  ■  The  Military  Health  System 
(MHS)  strives  to  provide  high-quality  care  and  improve 
outcomes  for  individuals  with  psychological  health  condi¬ 
tions.  Over  the  past  decade,  the  MHS  has  provided  care 
to  a  growing  number  of  individuals  with  psychological 
health  conditions,  such  as  post-traumatic  stress  disorder 
(PTSD)  and  major  depressive  disorder.  However,  little  is 
known  about  the  extent  to  which  the  MHS  delivers  care 
that  is  consistent  with  evidence-based  clinical  practice 
guidelines  or  whether  it  is  achieving  positive  outcomes  for 
its  service  members.  To  better  understand  these  issues,  the 
Defense  Centers  of  Excellence  for  Psychological  Health 
and  Traumatic  Brain  Injury  (DCoE)  asked  the  RAND 
Corporation  to  conduct  case  studies  to  identify  the  ways 
in  which  civilian  health  plans  measure  the  quality  of 
psychological  health  care  delivered  by  providers  in  their 
networks.  This  work  was  part  of  a  larger  RAND  effort 
to  develop  a  framework  and  identify  a  set  of  measures 
for  monitoring  the  quality  of  care  the  MHS  provides  for 
psychological  health  conditions. 
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MEASURING  THE  QUALITY  AND 
VALUE  OF  PSYCHOLOGICAL  HEALTH 
CARE 

Health  care  quality  is  a  concept  that  has  been  variously  defined 
in  the  literature.  For  example,  the  Institute  of  Medicine  (2001) 
has  described  quality  as  the  “degree  to  which  health  care 
systems,  services  and  supplies  for  individuals  and  populations 
increase  the  likelihood  for  positive  health  outcomes  and  are 
consistent  with  current  professional  knowledge,”  while  the 
World  Health  Organization  states  that  “quality  of  care  is  the 
level  of  attainment  of  health  systems’  intrinsic  goals  for  health 
improvement  and  responsiveness  to  legitimate  expectations  of 
the  population”  (Murray,  Lauer,  and  Evans,  2000).  Although 
these  definitions  of  quality  vary,  they  highlight  the  fact  that 
quality  refers  to  characteristics  of  and  actions  taken  by  provid¬ 
ers  that  contribute  to  patient  health.  This  concept  of  quality  is 
most  commonly  measured  across  three  domains  (Donabedian, 
2005): 

•  Structures  are  provider,  practice,  or  system  characteris¬ 
tics  that  are  associated  with  high  quality  of  care  and  may 
be  considered  a  necessary  but  insufficient  condition  for 
quality.  These  structures  might  include  the  presence  of 
key  technologies  (e.g.,  electronic  medical  record  system)  or 
characteristics  of  providers  and  staff  (e.g.,  mix  of  registered 
nurses  and  licensed  practical  nurses  on  a  nursing  unit). 
Structural  characteristics  are  also  relatively  concrete  and 
more  easily  assessed  compared  to  processes  of  care.  The 
decision  about  which  structural  characteristics  to  measure 
should  be  guided  by  the  extent  to  which  those  structures 
are  associated  with  processes  of  care  and  outcomes  of 
interest. 

•  Processes  of  care  refer  to  interventions  performed  on  and 
for  patients  and  are  often  operationalized  as  compliance 
with  evidence-based  care  guidelines.  Instead  of  measur¬ 
ing  the  outcomes  of  medical  care,  processes  of  care  can  be 
used  to  assess  the  extent  to  which  providers  deliver  “good” 
or  “appropriate”  medical  care.  Processes  of  care  variables 
reflect  the  actual  actions  of  a  health  care  provider  or  system 
and,  therefore,  may  be  more  appropriate  variables  in  com¬ 
paring  providers.  To  measure  processes  of  care,  however, 
health  care  systems  must  carefully  consider  which  processes 
of  care  are  most  appropriate  to  measure  and  the  extent  to 
which  those  processes  lead  to  the  outcomes  of  interest. 

•  Outcomes  are  heath  states  of  individuals  that  result  from 
the  delivery  of  health  care.  Using  outcomes  to  measure  the 
quality  of  medical  care  can  be  especially  useful  as  most 


stakeholders  agree  that  health  states,  such  as  recovery, 
restoration,  and  survival  from  illness,  are  the  end  goal  of 
medical  care,  and  outcomes  tend  to  be  concrete  and  validly 
measured  (e.g.,  death).  Outcomes,  however,  are  not  always 
the  direct  result  of  medical  care,  and  many  conditions  out¬ 
side  of  the  control  of  health  care  providers  (e.g.,  patients’ 
socioeconomic  status)  affect  outcomes,  rendering  the  com¬ 
parison  of  health  care  providers  difficult.  Furthermore,  it  is 
not  always  clear  which  outcomes  should  be  used,  and  many 
outcomes  can  be  challenging  to  measure. 

More  recently,  measurement  experts  have  identified  other 
domains  across  which  quality  of  care  might  be  measured.  Par¬ 
ticularly,  the  National  Quality  Measure  Clearinghouse,  a  pub¬ 
licly  available  repository  of  quality  measures  established  by  the 
Agency  for  Healthcare  Research  and  Quality,  includes  access 
to  care  and  patient  experience  as  key  quality  measure  domains 
(Agency  for  Healthcare  Research  and  Quality,  2014): 

•  Access  to  care  refers  to  the  extent  to  which  patients  can 
attain  timely  and  appropriate  care  from  health  care  provid¬ 
ers.  Access  to  care  can  be  measured  in  a  variety  of  ways, 
such  as  accessing  data  from  an  appointment  management 
system,  calling  a  provider  practice  to  assess  the  availability 
of  appointments,  or  estimating  the  number  of  providers  of 
a  certain  type  per  1,000  individuals. 

•  Patient  experience  refers  to  patient  reports  of  their  obser¬ 
vations  of  and  participation  in  the  health  care  system.  A 
patient  can  be  asked  to  report  on  structure  (e.g.,  timeli¬ 
ness  of  care),  process  of  care  (e.g.,  provider  listened  and 
respected  patient,  provider  discussed  treatment  options), 
and  outcomes  of  care  (e.g.,  patient’s  perception  of  the  help¬ 
fulness  of  treatment). 

Measurement  of  health  care  quality  is  difficult  regardless 
of  the  clinical  conditions  of  interest.  However,  the  systematic 
measurement  of  psychological  health  care  quality  is  particularly 
complex  relative  to  the  measurement  of  physical  health  care 
quality  (Addington  et  ah,  2005;  Hermann  et  ah,  2004;  Insti¬ 
tute  of  Medicine,  2006;  Kilbourne,  Keyser,  and  Pincus,  2010; 
Williams,  Cerese,  and  Cuny,  2007).  Many  of  these  challenges 
have  been  described  in  detail  in  the  Institute  of  Medicine’s 
report  Improving  the  Quality  of  Mental  Health  Care  and  Sub¬ 
stance  Abuse  (2006).  Key  barriers  described  in  the  report  are: 

•  Diverse  workforce:  Psychological  health  care  is  delivered 
by  a  variety  of  provider  types  working  in  a  number  of 
different  settings,  such  as  primary  care  practices,  therapist 


3 


offices,  prisons,  schools,  and  community  organizations. 
Providers  working  in  these  various  settings  have  very  dif¬ 
ferent  objectives,  clinical  approaches,  and  skills,  making 
systematic  measurement  across  groups  challenging. 

•  Unclear  locus  of  accountability:  Because  many  patients 
with  psychological  health  conditions  see  multiple  providers 
in  a  variety  of  settings,  it  is  challenging  to  determine  which 
providers  are  accountable  for  providing  various  processes  of 
care  for  such  patients. 

•  Different  types  of  evidence:  Compared  to  many  physi¬ 
cal  health  conditions,  a  significant  amount  of  the  evidence 
for  psychological  health  treatments  is  drawn  from  quasi- 
experimental  designs,  as  opposed  to  randomized  controlled 
trials.  Therefore,  wider  stakeholder  groups  might  view 
many  psychological  health  quality  measures  as  lacking  the 
requisite  evidence  to  be  considered  valid. 

The  concept  of  “value”  incorporates  both  quality  and 
cost  into  the  definition.  Value  has  been  defined  in  a  number 
of  different  ways  and,  importantly,  holds  different  meanings 
for  various  stakeholders.  However,  a  single,  unified  definition 
clearly  and  succinctly  defines  value  as  “outcomes  relative  to 
costs”  (Porter,  2010).  Because  value  incorporates  measures  of 
quality,  measuring  the  value  of  psychological  health  care  is  sus¬ 
ceptible  to  all  the  challenges  listed  above.  Furthermore,  because 
value  combines  both  outcomes  and  cost  data,  measuring  value 
requires  a  theoretical  and  computational  burden  that  is  much 
greater  than  that  for  developing  either  type  of  measure  alone. 

Despite  these  challenges,  health  plans  are  assessing  the 
quality  of  psychological  health  care  delivered  by  the  providers 
in  their  networks.  In  some  cases,  health  plans  may  be  using 
quality  and  value  measures  to  systematically  monitor  care 
delivered  by  the  providers  in  their  networks  and  to  inform  and 
monitor  quality  improvement  and  cost  reduction  efforts.  In 
other  cases,  health  plans  may  forgo  use  of  standardized  mea¬ 
sures  in  favor  of  other  means  to  assess  quality,  such  as  creden- 
tialing  processes,  peer  review,  or  tracking  patient  complaints. 
Health  plans  often  consider  efforts  to  assess  the  quality  of  care 
to  be  proprietary  (because  demonstrating  increased  capacity  for 
quality  measurement  and  improvement  is  considered  a  strategic 
advantage),  and  health  plans’  “best  practices”  in  this  domain 
are  generally  not  publicly  reported  in  academic  journals  or  gray 
literature  (e.g.,  trade  publications).  Therefore,  we  conducted 
a  small  set  of  case  studies  of  health  plans  to  describe  their 
approaches  to  assessing  the  quality  of  psychological  health  care. 
These  case  studies  had  three  primary  aims: 


The  sampling  frame 
for  this  study  included 
national  managed  care 
organizations,  integrated 
delivery  systems,  and 
behavioral  health 
managed  care  plans. 

1.  to  describe  health  plans’  efforts  to  assess  the  quality  of 
psychological  health  care  and  the  extent  to  which  quality 
measures  were  used  to  monitor  health  plan  performance 
and  guide  quality  improvement  strategies 

2.  to  describe  challenges  and  opportunities  related  to  health 
plans’  efforts  to  assess  the  quality  of  psychological  health 
care 

3.  to  describe  health  plans’  efforts  to  actively  assess  the 
value  of  psychological  health  care  and  the  extent  to 
which  value  measures  were  used  to  monitor  health  plan 
performance  and  guide  cost  reduction  efforts. 

The  overall  goal  of  the  case  studies  was  to  provide  the 
Department  of  Defense  (DoD)  with  information  related  to  how 
psychological  health  care  quality  and  value  are  being  assessed 
in  other  health  care  systems  to  inform  the  development  of  a 
comprehensive  measurement  system  for  psychological  health 
care  within  the  MHS. 


CASE  STUDY  METHODOLOGY 

The  sampling  frame  for  this  study  included  national  managed 
care  organizations  (MCO),  integrated  delivery  systems  (IDS, 
i.e.,  a  network  of  organizations  that  provide  a  full  continuum 
of  care,  such  as  physician  services,  hospital  care,  and  insurance 
coverage),  and  behavioral  health  managed  care  plans  (some¬ 
times  referred  to  as  “behavioral  health  carve-outs”  because 
they  are  contracted  to  provide  only  behavioral  health  benefits). 
RAND  chose  these  three  organization  types  because  they 
represent  the  three  primary  types  of  civilian-sector  payers  for 
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psychological  health  services.  The  case  study  participants  were 
chosen  based  on  convenience  sampling,  relying  on  RAND’s 
familiarity  with  organizations  that  pay  for  and  provide  psy¬ 
chological  health  services,  as  well  as  the  key  individuals  within 
those  organizations.  RAND  experts  nominated  organizations 
believed  to  be  leaders  in  quality  measurement  and  improvement 
within  the  civilian  health  care  sector  and  could  be  exemplar 
organizations.  The  selection  of  the  organizations  was  based  on 
personal  communication  and  experience  with  the  organiza¬ 
tions,  as  well  as  a  review  of  literature  in  which  these  organiza¬ 
tions’  work  was  highlighted.  Arguably,  the  U.S.  Department  of 
Veterans  Affairs  (VA)  would  be  the  most  analogous  organiza¬ 
tion  to  the  MHS.  However,  there  are  several  joint  DoD  and 
VA  efforts  that  facilitate  cross-agency  communication  (e.g.,  the 
VA/DoD  Integrated  Mental  Health  Strategy).  Therefore,  we 
believed  that  the  MHS  would  already  be  more  familiar  with 
initiatives  within  the  VA.  In  consultation  with  DCoE,  they 
agreed  with  our  strategy  to  sample  civilian  health  plans. 

We  then  contacted  executives  at  each  of  these  organizations 
to  assess  their  interest  in  participating  in  an  interview.  Of  the 
three  organizations  we  initially  selected,  all  participated.  For 
each  organization,  we  conducted  two  interviews  with  one  to 


two  representatives.  Our  goal  was  to  interview  mid-  to  high- 
level  executives  responsible  for  overseeing  performance  manage¬ 
ment,  clinical  programs,  and/or  analytics. 

The  first  plan  was  a  national  MCO,  where  we  conducted 
interviews  with  a  manager  for  quality  improvement  for  behav¬ 
ioral  health  and  a  senior  clinical  executive.  The  second  health 
plan  was  a  large  regional  IDS,  which  was  organized  as  a  health 
maintenance  organization  comprising  physicians,  hospitals,  and 
insurance  coverage.  We  chose  to  include  an  IDS  because  this 
was  the  organization  in  the  civilian  sector  most  similar  to  the 
MHS.  At  the  IDS,  we  interviewed  a  senior  clinical  director  for 
mental  health  and  substance  abuse  services  and  a  senior  execu¬ 
tive  for  quality  improvement  for  behavioral  health.  The  third 
health  plan  was  a  large,  national,  managed  behavioral  health 
organization  (MBHO),  where  we  interviewed  a  senior  executive 
for  quality  improvement  and  a  senior  executive  for  research. 

We  conducted  semistructured  interviews  with  each  of  the 
health  plan  representatives.  RAND  developed  an  interview  pro¬ 
tocol  based  on  the  content  areas  of  most  interest  to  DCoE  (i.e., 
how  civilian  health  plans  were  measuring  quality  and  value). 
The  protocol  covered  three  main  topics.  First,  we  asked  partici¬ 
pants  to  describe  their  health  plan  and  the  types  of  products 


Select  Types  of  Quality  Measures  Used  by  Health  Plans,  by  Data  Source,  as  Cited 
During  Interviews 

Administrative  data 

Process  measures  assessing  the  receipt  of  care  for  treatment  of  alcohol  and  drug  dependence  and  depression 
Process  measures  assessing  the  receipt  of  care  following  mental  health  hospitalizations 
Utilization  (e.g.,  medication  possession  ratio  and  hospitalizations) 

Readmissions 
Inpatient  length  of  stay 

Medical  record  reviews 

Appropriate  assessments 
Treatment  plans 

Interdisciplinary  communication 

Member  feedback 

Consumer  complaints 
Patient  experience 

Patient-reported  outcomes 

Changes  in  general  health 

Changes  in  psychological  health  symptoms 

Changes  in  drug  and  alcohol  use 
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offered.  Second,  we  asked  about  quality  measures  and  how  they 
were  used  to  monitor  performance  and  inform  quality  improve¬ 
ment  efforts  (e.g.,  How  would  you  define  “quality”  as  related 
to  psychological  health  care?  How  do  you  measure  the  quality 
of  psychological  health  services?  Which  measures  have  been 
the  easiest  or  hardest  to  implement?  Do  you  use  these  mea¬ 
sures  for  quality  improvement?).  Third,  we  asked  about  value 
measures  and  effects  of  these  measures  on  cost  reduction  efforts 
(e.g.,  How  do  you  measure  the  cost/efficiency  of  psychological 
health  care?  Which  measures  have  been  the  easiest  or  hardest 
to  implement?  Do  you  use  these  measures  to  monitor  cost  and 
efficiency?). 

Two  RAND  researchers  conducted  each  interview,  with 
one  leading  the  interview  and  the  second  taking  extensive 
notes.  All  interviews  were  conducted  by  phone.  All  participants 
provided  oral  informed  consent.  Participants  were  told  that 
the  information  gathered  in  the  interview  would  be  used  for 
research  purposes  only  and  that  their  responses  would  be  kept 
confidential  but  that  readers  might  be  able  to  identify  organiza¬ 
tions  or  interviewees  by  inference.  After  the  completion  of  the 
interviews,  the  two  researchers  used  a  conventional  content 
analysis  (Hsieh  and  Shannon,  2005)  to  extract  themes  directly 
from  typed  notes.  The  researchers  independently  identified  (1) 
how  health  plans  were  measuring  quality  and  value,  and  (2)  any 
themes  that  were  similar  across  the  interviews.  The  researchers 
then  met  to  discuss  their  findings,  identify  common  themes 
across  the  interviews,  and  reach  a  consensus  on  any  thematic 
discrepancies. 


PSYCHOLOGICAL  HEALTH  QUALITY 
MEASUREMENT  IN  HEALTH  PLANS  IS 
COMMON 

Participants  reported  that  they  regularly  measured  and  tracked 
psychological  health  quality  measures  using  four  key  data 
sources  (see  box): 

1.  administrative  data 

2.  medical  record  review 

3.  member  feedback 

4.  patient-reported  outcomes. 


Administrative  Data 

Administrative  data  are  records  of  services  provided  to 
patients,  which  are  generally  collected  for  the  purposes  of  bill¬ 


ing  or  registration.  The  data  can  include  records  of  what  service 
was  delivered,  any  diagnoses  related  to  the  service,  and  service 
dates. 

All  the  health  plans  reported  measuring  and  tracking  qual¬ 
ity  using  administrative  data-based  measures.  In  this  context, 
administrative  data  were  generally  used  to  construct  Healthcare 
Effectiveness  Data  Information  Set  (HEDIS)  measures  and 
HEDIS-like  measures  (e.g.,  measures  similar  to  HEDIS  but 
using  different  measure  specifications).  HEDIS  measures  are 
constructed  using  health  insurance  claims  data  and  used  to 
measure  the  performance  of  health  plans  (National  Commit¬ 
tee  for  Quality  Assurance  [NCQA],  2013).  In  particular,  the 
civilian  health  plans  that  we  interviewed  routinely  implemented 
HEDIS  measures  required  for  NCQA  accreditation.  NCQA  is 
a  nationally  recognized  private  501(c)(3)  not-for-profit  organi¬ 
zation  that  offers  various  levels  of  health  plan,  physician,  and 
other  organizational  designations  that  include  accreditation, 
certification,  and  recognition  (NCQA,  2012).  For  accredita¬ 
tion,  health  plans  are  required  to  track  and  report  the  use  of 
evidence-based  care,  particularly  antidepressant  medication 
management,  follow-up  after  hospitalization  for  mental  illness, 
identification  of  alcohol  and  other  drug  services,  initiation 
and  engagement  of  alcohol  and  other  drug  dependence  treat¬ 
ment,  follow-up  care  for  children  prescribed  attention  deficit 
hyperactivity  disorder  (ADHD)  medication,  and  mental  health 
utilization  (among  other  nonpsychological  health  measures). 
Because  its  performance  was  reported  through  NCQA,  the  IDS 
monitored  its  own  performance  on  a  regular  basis  and  strove  to 
be  at  the  90th  percentile  or  above  on  each  measure.  However, 
other  plans  did  not  report  having  a  regular  schedule  to  moni¬ 
tor  their  performance  on  HEDIS  measures  or  to  set  specific 
performance  goals. 

The  IDS  was  also  required  to  report  similar  measures 
related  to  access  to  psychological  health  care  services  to  a  state 
regulatory  agency.  These  measures  included  the  extent  to  which 
members  have  access  to: 

•  a  nonurgent  psychological  health  provider  within  ten  days 
of  a  request  for  an  appointment 

•  nonurgent  specialist  care  within  15  business  days  of  a 
request  for  an  appointment 

•  urgent  care  within  48  hours  of  a  request  for  an 
appointment. 

Other  administrative  data-based  measures  tracked  by 
the  health  plans  included  readmissions,  length  of  stay,  dura¬ 
tion  between  admissions,  and  medication  possession  ratio. 
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The  MCO  and  MBHO  stated  that  they  preferred  developing 
administrative  data-based  measures,  such  as  medication  pos¬ 
session  ratio,  because  the  requisite  data  are  available  in  claims 
and  do  not  require  collecting  primary  data  from  patients  or 
providers. 

Medical  Record  Reviews 

Two  of  the  health  plans  reported  performing  regular 
reviews  of  patient  medical  records  to  assess  both  the  accuracy 
and  sufficiency  of  provider  documentation.  These  audits  were 
part  of  the  process  for  credentialing  providers  and  were  often 
performed  in  an  ad  hoc  manner.  While  these  efforts  did  not 
yield  data  to  construct  quality  measures  per  se,  the  health  plans 
reported  that  such  efforts  were  an  important  part  of  assessments 
of  the  quality  of  care.  The  MCO  conducted  medical  record 
reviews  with  a  random  selection  of  60-80  psychological  health 
providers  (including  primary  and  specialty  care)  per  year  (out 
of  120,000  network  providers)  and  examined  these  records  for 
documentation  of  appropriate  assessments  (e.g.,  assessing  for 
homicide  and/or  suicide,  measuring  patient-reported  outcomes), 
follow-up  after  a  specific  diagnosis  (e.g.,  substance  abuse), 
discharge  plans,  and  other  types  of  clinical  care.  In  all,  about 
40  items  were  examined,  and  every  provider  received  feedback 
about  these  audits  (described  in  more  detail  below).  The  IDS 
also  conducted  medical  record  reviews  and  specifically  men¬ 
tioned  monitoring  progress  notes  and  specific  quality  measures 
tied  to  quality  improvement  initiatives.  The  IDS  randomly 
selected  two  patient  intake  visits  and  three  follow-up  visits 
for  each  provider  for  review  to  ensure  that  patient  outcomes 
were  routinely  measured,  treatment  plans  were  present  in  the 
progress  notes,  and  reminders  were  sent  to  patients  about  how 
they  could  communicate  with  their  provider  if  crisis  services 
were  needed.  An  important  aspect  of  the  IDS’s  medical  record 
review  was  to  examine  coordination  of  care  and  whether 
interdisciplinary  communication  was  occurring  within  the 
IDS  (e.g.,  whether  primary  care  physicians  and  psychiatrists 
were  communicating  about  a  patient).  At  least  once  a  year, 
chairs  from  each  department  across  the  country  discussed  these 
reviews  and  other  quality  improvement  efforts  as  part  of  their 
quality  improvement  committee.  The  MBHO  did  not  discuss 
the  use  of  medical  record  reviews  as  a  key  aspect  of  its  quality 
assessment  efforts. 

Respondents  cited  the  medical  record  review  as  an  impor¬ 
tant  aspect  of  their  efforts  to  assess  and  improve  the  quality  of 
care  delivered  by  providers  in  their  network.  However,  these 


efforts  did  not  yield  quality  measures  that  could  be  systemati¬ 
cally  tracked,  and  none  of  the  plans  conducted  record  reviews 
to  routinely  assess  patient  outcomes. 

Member  Feedback 

All  the  health  plans  reported  collecting  patient  feedback 
in  the  form  of  consumer  complaints  and/or  patient  experience 
surveys.  For  example,  the  MCO  and  IDS  reported  that  they 
regularly  collected,  tracked,  and  followed  up  on  complaints 
about  individual  providers  (e.g.,  if  patients  reported  mistreat¬ 
ment,  privacy  breaches,  or  that  an  office  was  not  conducive  to 
therapy).  In  addition,  the  IDS  and  MBHO  reported  adminis¬ 
tering  a  patient  or  member  survey  to  track  service  satisfaction. 
The  IDS  reported  using  a  member  survey,  which  included 
15-20  items  related  to  satisfaction  (topics  included  ratings  of 
treatment  results  and  interactions  with  providers).  Surveys  were 
designed  to  be  administered  by  providers  and  their  administra¬ 
tive  staffs  when  patients  arrived  for  appointments.  As  part  of 
their  health  plans’  general  NCQA  accreditation,  the  MCO  and 
IDS  also  were  required  to  measure  and  report  patient  experi¬ 
ence  using  the  Consumer  Assessment  of  Healthcare  Providers 
and  Systems  instrument  (Agency  for  Healthcare  Research  and 
Quality,  2012).  However,  these  efforts  were  not  specifically 
mentioned  during  the  interviews  as  a  key  part  of  their  efforts  to 
measure  and  monitor  the  quality  of  psychological  health  care. 

Patient- Reported  Outcomes 

Two  of  the  health  plans  reported  collecting  patient- 
reported  outcomes  and  that  these  measurements  helped  with 
several  quality  initiatives.  The  IDS  administered  a  patient 
survey  that  assessed  symptom  improvement  (e.g.,  the  nine- 
item  Patient  Health  Questionnaire  (PHQ-9)  with  four  addi¬ 
tional  items  that  assess  anxiety  and  general  functioning).  This 
health  plan  reported  asking  providers  to  administer  an  assess¬ 
ment  within  the  first  two  sessions  and,  four  months  later,  to 
assess  depression  symptoms  (PHQ-9;  Spitzer,  Kroenke,  and 
Williams,  1999),  mental  and  physical  functioning  (Medical 
Outcomes  Survey  Short  Form)  (Ware,  Kosinski,  and  Keller, 
1996),  substance  use  (CAGE-AID)  (Brown  and  Rounds,  1995), 
and  other  clinical  symptoms.  The  MBHO  asked  providers  to 
administer  a  patient  assessment  that  included  items  from  the 
Symptom  Checklist-90  (Derogatis,  1977),  CAGE-AID  Health 
Performance  Questionnaire  (Brown  and  Rounds,  1995),  Short 
Form  (SF)-36  (Ware,  2003),  and  other  questions  regarding 


All  of  the  health  plans  stated  that  collection  of  patient- 
reported  outcomes  data  was  arguably  the  most  difficult 
aspect  of  their  quality  measurement  initiatives. 
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comorbid  conditions  and  utilization  of  medical  services  in  the 
previous  six  months.  The  MBHO  asked  providers  to  admin¬ 
ister  this  assessment  within  the  first  two  visits  to  the  provider. 
The  provider  was  then  expected  to  enter  this  information  into 
an  online  portal  or  fax  it  to  the  MBHO.  After  four  months, 
the  MBHO  mailed  a  follow-up  assessment  tool,  containing 
the  same  items,  to  the  patient  to  fill  out  and  return.  Both  the 
IDS  and  the  MBHO  reported  that  these  data  were  designed 
to  inform  providers  about  how  their  patients  were  progressing 
and  to  identify  at-risk  patients.  For  example,  providers  from  the 
MBHO  could  log  onto  a  portal  to  obtain  feedback  about  how 
their  clients  were  improving  or  worsening  and  also  had  high- 
risk  clients  flagged.  However,  it  is  not  clear  whether  providers 
actually  use  this  information.  Both  health  plans  also  reported 
that  data  collection  from  patients  was  challenging  because  of 
poor  provider  and  patient  response  rates. 


HEALTH  PLANS  PRIMARILY  FOCUSED 
ON  COLLECTING  AND  REPORTING 
MEASURES  OF  THE  PROCESS  OF  CARE 

The  health  plans  that  we  interviewed  were  actively  assessing  the 
quality  of  care  delivered  by  providers  in  their  networks.  How¬ 
ever,  the  quality  measurement  efforts  were  relatively  limited 
in  scope  and  scale  and  may  not  have  provided  a  comprehen¬ 
sive  view  of  quality.  The  measures  that  were  most  consistently 
reported  assessed  processes  of  care,  such  as  documentation  of 
appropriate  symptom  assessments  or  rates  of  follow-up  after 
hospitalizations.  These  process  measures  were  generally  limited 
in  scope,  however,  focusing  on  drug  and  alcohol  dependence, 
depression,  and  posthospitalization  care,  and  were  often 
assessed  on  a  rather  small  scale  (e.g.,  medical  record  audits 
of  a  small  number  of  providers).  Health  plans  rarely  reported 
assessing  the  structure  of  psychological  health  care  across  their 
networks.  Health  plans  reported  the  use  of  access  measures  to 


inform  whether  provider  networks  were  compliant  with  state 
regulatory  requirements  (e.g.,  availability  of  appointments). 

While  health  plans  were  enthusiastic  about  the  use  of 
patient-reported  outcome  measures,  their  efforts  were  still  in 
the  nascent  stages  as  they  continue  to  examine  opportunities 
to  engage  patients  and  providers  in  the  data  collection  process 
by  helping  them  see  value  in  having  such  data  available.  Health 
plans  routinely  tracked  such  measures  as  consumer  complaints 
and  patient  satisfaction,  which  may  be  proxies  for  factors 
associated  with  the  quality  of  care.  For  example,  consumer 
complaints  likely  reflect  the  fact  that  providers  do  not  deliver 
processes  of  care  appropriately  for  their  patients  or  that  there 
are  problems  with  some  aspect  of  the  structure  of  and  access 
to  care  (e.g.,  availability  of  appointments,  cleanliness  of  office 
space). 


HEALTH  PLANS  USE  QUALITY 
MEASURES  TO  IMPROVE  THE  QUALITY 
OF  PSYCHOLOGICAL  HEALTH  CARE 

Health  plans  used  quality  measures  in  a  variety  of  ways.  First, 
all  health  plans  reported  using  administrative  data-based 
measures  to  plan  and  monitor  quality  improvement  efforts.  For 
example,  the  health  plans  often  used  their  own  care  manage¬ 
ment  services  to  help  ensure  that  their  beneficiaries  received 
recommended  care.  The  IDS  undertook  a  number  of  initiatives 
to  improve  HEDIS  scores,  such  as  having  in-house  care  manag¬ 
ers  schedule  outpatient  follow-up  visits  while  patients  were  still 
in  the  hospital  to  improve  aftercare  rates.  They  also  monitored 
“high-risk”  patients  on  discharge  and  reviewed  their  cases  to 
ensure  that  they  had  a  follow-up  appointment  within  seven 
days  of  discharge  (and  called  the  patient  and  provider  if  not). 
The  MBHO  used  claims  data  (e.g.,  utilization,  comorbidities, 
hospitalizations)  to  flag  high-risk  patients.  The  plan  then  sent 
associated  alerts  to  providers  and  offered  additional  outreach  to 
members  when  these  types  of  flags  were  identified.  In  addition, 
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the  plan  tracked  gaps  in  care  in  certain  process  measures,  such 
as  medication  prescribing,  as  a  sign  that  patients  may  be  at  high 
risk  for  rehospitalization. 

Health  plans  also  reported  that  an  important  part  of  their 
quality  improvement  strategies  was  to  provide  feedback  to 
providers  and  facilities  on  their  performance.  For  example,  the 
MCO  reported  using  medical  record  reviews  to  provide  feed¬ 
back  to  providers.  If  any  of  the  60-80  providers  whose  patients’ 
medical  records  were  reviewed  received  an  overall  score  below 
80  percent,  the  plan  instituted  a  corrective  action  plan.  Provid¬ 
ers  generally  improved  when  reaudits  were  conducted  because, 
if  they  did  not  comply,  the  health  plan  removed  them  from 
the  network.  The  MCO  also  monitored  patient  complaints; 
provider  recredentialing  could  be  affected  if  there  were  three  or 
more  complaints.  The  MCO  has  considered  sending  providers 
reports  on  their  performance  compared  to  peer  physicians  but 
has  not  implemented  this  strategy  yet  because  initial  efforts 
were  unsuccessful.  Representatives  from  the  MCO  noted  that 
their  previous  attempts  to  do  this  were  impeded  by  difficulties 
communicating  with  some  physicians  who  were  not  connected 
through  electronic  records  and  that  physicians  with  “low  per¬ 
formance”  assumed  the  data  were  wrong  and  did  not  trust  the 
data.  Finally,  the  MCO  is  also  examining  ways  to  provide  feed¬ 
back  to  facilities  regarding  patients’  length  of  stay  compared  to 
those  of  competitors. 

The  IDS  reported  using  patient-reported  outcomes  to 
understand  how  well  each  location  and  provider  managed  care. 
This  information  was  then  used  to  either  target  resources  to 
providers  that  needed  more  support  or  to  identify  providers  that 
were  top  performers.  The  MBHO  enabled  providers  to  view 
their  patient-reported  outcomes  compared  to  organizational 
benchmarks  or  other  clinical  thresholds  for  certain  psychologi¬ 
cal  health  conditions.  This  online  portal  was  tied  to  the  billing 
portal  system  to  encourage  providers  to  use  the  platform. 

Patient  outcomes  were  also  connected  to  patient  claims  data  to 
help  providers  identify  and  monitor  high-risk  patients. 

Finally,  health  plans  were  starting  to  use  performance  data 
to  implement  “value-based  insurance  designs,”  wherein  health 
plans  provide  information  and  incentives  to  encourage  patients 
to  choose  “high-performing”  providers  (Chernew,  Rosen,  and 
Fendrick,  2007).  High-performing  providers  are  selected  using 
either  cost  or  quality  metrics  or  a  combination  of  the  two 
depending  on  the  goals  of  the  insurance  design.  Such  strategies 
are  often  referred  to  as  “steering”  (Scanlon  and  Martsolf,  2009). 
The  MCO  was  beginning  to  use  quality  measures  to  steer 
clients  toward  preferred  drug  and  alcohol  programs  that  report 


better  outcomes.  The  strategy  included  ranking  each  facility  by 
key  outcomes,  such  as  readmission  rates  and  cost  of  care.  The 
health  plan  would  then  provide  members  with  incentives  (i.e., 
reduced  copay)  to  visit  those  providers.  Such  steering  programs 
do  face  significant  challenges.  Particularly,  many  providers 
remain  unconvinced  that  methodological  approaches  used 
to  categorize  providers  are  valid  and  reliable,  leading  to  con¬ 
cerns  that  some  providers  would  be  unfairly  penalized  in  such 
a  scheme.  In  fact,  there  have  been  a  number  of  high-profile 
court  cases  attempting  to  block  efforts  by  insurance  companies 
to  implement  such  steering  programs  (Scanlon  and  Martsolf, 
2009). 


MEASURING  THE  QUALITY  OF 
PSYCHOLOGICAL  HEALTH  CARE  IS 
CHALLENGING 

Health  plans  reported  a  number  of  challenges  to  measur¬ 
ing  the  quality  of  care  delivered  by  their  network  providers. 

All  the  health  plans  stated  that  collection  of  patient-reported 
outcome  data  was  arguably  the  most  difficult  aspect  of  their 
quality  measurement  initiatives.  The  health  plans  all  stated  that 
collection  of  these  data  was  especially  hampered  by  provider 
resistance.  Many  of  these  providers  have  argued  that  psycho¬ 
logical  health  is  subjective,  individual-based,  and  not  “one  size 
fits  all”;  therefore,  having  objective  outcome  standards  was  not 
reasonable.  The  health  plan  representatives  also  stated  that  pro¬ 
viders  were  not  typically  trained  to  monitor  the  quality  of  their 
services  and  were  resistant  to  measuring,  collecting,  and  report¬ 
ing  these  data.  Furthermore,  for  many  providers,  only  a  small 
fraction  of  their  patients  was  covered  by  any  given  health  plan. 
Thus,  providers  did  not  see  the  value  of  completing  additional 
paperwork  to  document  quality  and  value  measurement  for  a 
single  health  plan.  Also,  some  providers  felt  that  collecting  and 
entering  patient-level  data  interfered  with  patient  care.  Echoing 
these  reported  challenges  for  providers,  the  literature  suggests 
that  collecting  and  using  patient-reported  data  is  difficult.  For 
example,  while  one  study  found  that  74  percent  of  psychologi¬ 
cal  health  systems  collect  patient-reported  data  (Schoenwald 
et  ah,  2008),  another  study  found  that  few  providers  actually 
use  this  information  in  their  clinical  practice  (Garland,  Kruse, 
and  Aarons,  2003). 

To  overcome  provider  reluctance  related  to  collecting 
patient-reported  outcome  data,  the  MCO  was  piloting  a  pro¬ 
gram  with  three  to  four  provider  groups  in  different  parts  of  the 
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country  to  pay  providers  for  using  evidence-based  assessments 
at  the  beginning  of  treatment  and  then  again  three  months 
later.  The  goal  of  this  effort  was  to  incentivize  providers  to 
conduct  these  assessments  and  monitor  clinical  outcomes  over 
time.  The  providers  asked  four  questions  about  the  clinical 
status  of  the  patient  at  the  beginning  of  the  session  and  four 
questions  about  the  process  of  therapy.  Providers’  performance 
was  then  compared  to  a  national  database. 

Health  plans  stated  that  claims  records,  such  as  receipt  of 
certain  lab  tests,  were  much  easier  to  collect  but  often  lacked 
substantive  information.  For  example,  the  IDS  reported  that 
administrative-based  measures  (e.g.,  HEDIS  measures)  were 
easier  to  implement  because  the  data  were  available  in  claims, 
did  not  need  to  be  collected  separately,  and  did  not  rely  on  pro¬ 
vider  compliance  with  data  collection  requests.  However,  these 
measures  are  limited  because  they  focus  on  what  services  were 
delivered  by  providers,  as  opposed  to  patient  outcomes  (e.g., 
it  was  easy  to  know  whether  a  prescription  was  filled  but  not 
whether  the  patient  got  better).  Furthermore,  the  health  plans 
reported  that  complete  claims  data  were  often  unavailable  for 
three  months  or  more  after  the  date  of  service. 

Health  plans  noted  several  recommendations  that  could 
help  contribute  to  successful  quality  measurement,  including 
using  measures  already  established  in  the  literature,  creating  an 
organizational  climate  supportive  of  quality  measurement  (e.g., 
a  video  of  members  and  providers  sharing  their  experiences  and 
emphasizing  the  importance  of  measurement),  and  utilizing 
provider  champions  to  help  increase  provider  buy-in.  Finally, 
two  health  plans  reported  that,  when  measures  were  integrated 
into  providers’  clinical  operations  (e.g.,  electronic  health  record 
system),  the  data  were  easier  to  collect. 


MEASURING  THE  VALUE  OF 
PSYCHOLOGICAL  HEALTH  CARE  IS 
MORE  DIFFICULT  AND  LESS  COMMON 

The  participants  in  our  study  commented  significantly  less  on 
value  measurement  compared  to  quality  measurement.  This  was 
driven  both  by  the  fact  that  many  of  the  participants  worked  in 
areas  directly  related  to  quality  of  care  and  by  the  lesser  devel¬ 
opment  of  their  health  plans’  efforts  to  systemically  measure 
value  relative  to  quality.  Nonetheless,  health  plans  reported  that 
they  did  monitor  and  track  utilization  patterns  among  their 
beneficiaries.  For  example,  the  MCO  reported  monitoring  such 
measures  as  admissions  per  1,000  members,  hospital  days  per 


1,000  members,  and  length  of  stay,  while  the  MBHO  tracked 
access  rates,  diagnosis  rates,  and  changes  in  utilization  by  facil¬ 
ity. 

There  was  noticeable  variation  in  the  extent  to  which  the 
plans  measured  the  actual  costs  of  care  among  members  with 
psychological  health  diagnoses.  The  MCO  reported  regularly 
tracking  actual  costs  using  a  “per  member  per  month  (PMPM)” 
measure.  The  plan  produced  reports  on  PMPM  trends  on  a 
monthly  basis,  which  were  broken  out  by  a  number  of  cat¬ 
egories,  such  as  chemical  dependency  versus  mental  health, 
residential  versus  outpatient,  pharmacy  types,  and  patient  age. 
The  MBHO  also  tracked  PMPM  costs  but  did  not  provide 
specific  information  on  how  these  costs  were  used  or  dissemi¬ 
nated  across  the  institution.  Instead,  it  reported  more  regularly 
tracking  utilization  rates.  The  IDS  did  not  produce  any  regular 
reports  of  its  cost  of  care.  This  is  largely  explained  by  the  fact 
the  IDS  was  fully  integrated,  so  it  functioned  primarily  on  a 
fixed-cost  basis  and  did  not  pay  fees  for  services.  Therefore, 
marginal  service  costs  were  more  difficult  to  track.  The  direc¬ 
tor  of  mental  health  of  that  plan  reported  that  he  could  know 
the  fixed  costs  associated  with  the  psychological  health  delivery 
system  but  could  not  estimate  marginal  costs  or  efficiency.  IDS 
did  track  utilization  and  noted  that  a  long-term  goal  was  to  be 
able  to  use  this  utilization  data  to  produce  estimates  of  effi¬ 
ciency  and  answer  such  questions  as  “How  many  visits  does  it 
take  to  achieve  a  certain  outcome?”  He  believed  that  once  IDS 
was  able  to  obtain  enough  patient-reported  outcome  data,  it 
might  be  able  to  do  some  of  this  analysis. 

The  health  plans  that  did  track  and  monitor  utilization  and 
costs  used  these  metrics  for  a  number  of  different  purposes.  For 
example,  the  MCO  monitored  costs  and  utilization  closely  and, 
when  there  were  significant  changes  in  these  metrics,  deployed 
staff  for  greater  oversight  in  that  area.  Similarly,  the  MBHO 
analyzed  utilization  and  cost  data  to  better  understand  trends 
and  patterns  to  determine  whether  these  changes  were  occur¬ 
ring  at  specific  facilities  or  among  specific  demographic  groups. 
Then,  the  staff  would  work  with  facilities  to  understand  these 
spikes  in  utilization  (e.g.,  whether  patients  are  being  referred 
to  lower  levels  of  care)  and  to  create  initiatives  to  reduce  costs. 
However,  the  interviewees  acknowledged  that  measuring  value 
and  reducing  cost  based  on  these  measurements  was  challeng¬ 
ing  and  something  at  which  they  were  less  skilled. 


io 


IMPLICATIONS  FOR  PSYCHOLOGICAL 
HEALTH  QUALITY  AND  VALUE 
MEASURES  WITHIN  THE  MHS 

In  this  section,  we  consider  how  the  findings  from  these  case 
studies  can  inform  future  efforts  to  measure  and  improve  qual¬ 
ity  and  value  of  psychological  health  care  in  the  MHS. 

Continue  and  Expand  Tracking  of  Access 
and  Process  of  Care  Measures 

The  health  plans  focused  first  on  generating  measures  based 
on  readily  available  data.  Specifically,  plans  were  generating 
administrative  data-based  measures  of  access  to  care  (i.e., 
appointment  availability)  and  care  processes  (i.e.,  the  clini¬ 
cal  procedures  provided).  Like  the  civilian  health  plans  that 
we  interviewed,  the  MHS  routinely  collects  similar  data  as 
well.  For  example,  the  MHS  regularly  tracks  its  performance 
on  18  HEDIS  measures  within  the  direct  care  system  and  12 
measures  within  the  purchased  care  system  (Department  of 
Defense,  2014a;  Department  of  Defense,  2014b).  Four  of  these 
measures  are  related  to  psychological  health. 

The  civilian  health  plans,  however,  continue  to  expand 
their  collection  of  access  and  process  measures,  and  MHS 
might  do  likewise.  Specifically,  MHS  might  continue  to 
expand  measurement  related  to  psychological  health.  MHS 
could  improve  on  its  measurement  of  quality  by  ensuring  that 
such  measures  are  collected  systematically  across  all  military 
treatment  facilities  (MTFs)  and  services  and  that  measures  are 
regularly  reevaluated  to  include  additional  measures  that  may 
be  of  relevance  to  the  MHS.  While  some  MTFs  may  collect 
several  psychological  health  quality  measures,  very  few  mea¬ 
sures  are  collected  and  assessed  consistently  across  all  MTFs, 
services,  and  for  direct  and  purchased  care  (Department  of 
Defense,  2013,  2014c).  Collecting  different  measures  of  quality 
makes  it  challenging  to  compare  across  MTFs  and  care  systems. 
A  recent  RAND  report  provides  an  extensive  list  of  potential 
quality  measures  that  could  be  considered.  This  report  presents 
a  list  of  58  process,  structure,  and  outcome  measures  that  are 
candidates  to  be  used  to  assess  the  quality  of  PTSD  and  MDD 
treatment  (Hepner  et  ah,  2015).  An  important  step  in  expand¬ 
ing  the  scope  of  the  quality  measurement  is  to  update  and 
expand  DoD  policies  related  to  the  collection  and  assessment 
of  psychological  health  quality  measures.  If  the  MHS  were  to 
expand  its  performance  assessment  capabilities,  it  would  collect 


more  quality  information  than  many  civilian  health  plans  cur¬ 
rently  track  and  assess. 

Engage  Providers  in  the  Process 

Health  plans  reported  that  clinician  autonomy  often  acted  as  a 
large  barrier  to  quality  improvement  efforts.  Interviewees  and 
other  literature  report  that  provider  buy-in  is  critical  for  suc¬ 
cessful  implementation  of  any  quality  measurement  program 
(Addington  et  al.,  2005;  Baars  et  al.,  2010;  Baker,  1998;  Baker, 
1999;  Bremer  et  al.,  2008;  Gaebel  et  al.,  2012;  Kilbourne, 

Keyser,  and  Pincus,  2010;  Stein  et  al.,  2010;  Reich  et  al.,  2003). 
Particularly,  while  the  health  plans  we  interviewed  were  very 
enthusiastic  about  measuring  patient-reported  outcomes,  they 
reported  that  providers  were  reticent  to  collect  such  informa¬ 
tion.  To  ensure  adequate  implementation  of  such  measurement 
efforts,  health  plans  must  provide  a  compelling  case  to  providers 
that  the  use  of  such  data  is  personally  or  clinically  beneficial. 

For  example,  the  MCO  that  we  interviewed  has  provided  bonus 
payments  to  providers  for  using  the  patient-reported  outcome 
measure  tool.  However,  the  initiative  is  in  its  early  stages,  and  the 
plan  has  been  unable  to  assess  the  impact  of  this  incentive  on  the 
use  of  the  measure.  In  other  settings,  such  pay-for-performance 
mechanisms  have  been  used  in  an  attempt  to  improve  process  of 
care  and  outcome  measures  (Damberg  et  al.,  2014).  Health  plans 
have  also  considered  providing  incentives  to  patients  for  filling 
out  the  tools,  sending  the  information  to  the  health  plan,  and 
discussing  them  with  the  providers.  Likewise,  in  order  to  expand 
beyond  the  measurement  of  administrative  data-based  access 
and  processes  of  care  measures,  MHS  would  need  to  more  effec¬ 
tively  engage  MTF  providers  and  have  the  flexibility  to  require 
purchased  care  contractors  to  collect  and  report  quality  data. 

While  financially  incentivizing  providers  within  the  MHS 
may  not  be  feasible,  the  MHS  may  use  other  nonfinancial 
incentives,  such  as  mandates,  to  encourage  provider  compli¬ 
ance.  For  example,  through  an  explicit  policy  memo,  the 
military  is  encouraging  the  use  of  the  Behavioral  Health  Data 
Portal  (BDHP)  to  monitor  symptoms  for  patients  with  PTSD, 
depression,  and  anxiety  disorders  (Department  of  Defense, 
2014a;  Brewin,  2013).  These  data  could  populate  outcome 
measures  in  the  future.  Further,  the  MHS  might  work  with  the 
DoD  to  mandate  clinics  to  collect  quality  of  care  data.  Alter¬ 
natively,  the  MHS  might  formally  recognize  high-performing 
MTFs  and  providers  or  publicly  report  MTF  performance. 

This  increase  in  transparency  was  one  of  the  key  recommenda- 
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tions  in  a  recent  report  on  MHS  care  (Department  of  Defense, 
2014a). 

The  MHS  might  also  experiment  with  financial  and  other 
incentives  for  purchased  care  providers.  The  MHS  might  con¬ 
sider  incentivizing  purchased  care  providers  to  collect  quality 
data  and  participate  in  data  collection  efforts,  consistent  with 
recommendations  provided  in  the  Military  Health  System 
Review  Report  to  the  Secretary  of  Defense  (Department  of 
Defense,  2014a).  Incentives  could  be  incorporated  into  con¬ 
tracts  with  purchased  care  providers  or  through  preferential 
referrals  for  providers  who  participate  in  data  collection  and 
sharing.  For  providers  at  MTFs,  MHS  could  use  other  non- 
financial  means  to  encourage  provider  participation  in  the 
collection  of  quality  data  not  available  in  administrative  data, 
especially  patient-reported  outcomes.  Specifically,  the  MHS 
could  work  with  providers  to  demonstrate  that  the  use  of  qual¬ 
ity  measurement  systems  can  help  them  deliver  better  care  to 
their  patients.  A  number  of  studies  have  demonstrated  that 
patient-reported  outcome  tools  can  be  used  to  engage  patients 
and  families  in  the  care  process,  which  can,  in  turn,  improve 
outcomes  (Azocar  et  al.,  2003;  Brodey  et  al.,  2005;  Duffy  et  al., 
2008). 

Implement  Measures  of  Costs  and 
Utilization,  Which  Provide  an  Essential 
Foundation  for  Assessing  Value 

Although  the  health  plans  we  interviewed  expressed  interest 
in  measuring  the  value  of  psychological  health  care,  a  number 
of  significant  barriers  to  implementation  exist.  Value  measures 
remain  underdeveloped  for  all  health  conditions  but  especially 
for  psychological  health.  Some  have  suggested  calculating  the 
“value”  of  depression  care  by  estimating  the  cost  required  to 
achieve  a  given  outcome  (e.g.,  quality-adjusted  life  years  or 
depression-free  days).  However,  these  measures  require  the 
collection  of  valid  and  reliable  patient-reported  outcomes  over 
time.  As  discussed  earlier,  efforts  to  collect  such  outcome  data 
within  health  plans  have  been  extremely  time  consuming  and 
require  substantial  engagement  by  providers.  Given  the  strug¬ 
gles  of  civilian  health  plans  to  collect  outcomes  data,  the  MHS 
might  be  a  leader  in  developing  value  measures,  given  they 
are  beginning  to  see  success  systematically  collecting  patient 


outcome  data  through  such  programs  as  the  BDHR  However, 
to  move  toward  developing  measures  of  value,  the  MHS  might 
consider  regularly  measuring  cost  and  utilization  data.  Such 
data  could  be  linked  to  the  growing  outcome  measures  to 
create  measures  of  value.  The  first  step  toward  measuring  cost 
would  be  to  consistently  track  utilization  data,  readily  avail¬ 
able  in  claims,  which  include  hospitalizations,  readmissions, 
and  length  of  stay,  as  well  as  utilization  of  prescription  drugs 
and  ambulatory  care  visits.  Although  these  are  not  explicitly 
measures  of  cost,  they  provide  a  useful  starting  point  for  the 
MHS  to  move  toward  measuring  the  cost  and  value  of  care  it 
provides. 


LIMITATIONS 

The  case  studies  have  at  least  three  limitations.  First,  we  relied 
on  RAND  experts  to  nominate  health  plans  known  for  their 
quality  measurement  efforts.  This  process  for  selecting  organi¬ 
zations  was  neither  formal  nor  unbiased,  but  we  were  purpo¬ 
sive  in  this  regard  because  we  wanted  to  better  understand 
what  leading  health  plans  were  doing  and  how  their  efforts 
could  inform  DoD.  Second,  the  comments  provided  by  the 
two  representatives  from  each  organization  may  not  represent 
the  larger  views  and  efforts  of  the  entire  health  plan.  Finally, 
because  we  interviewed  only  three  health  plans,  the  results 
gathered  in  this  report  are  not  necessarily  generalizable  to  all 
civilian  health  plans. 


CONCLUSIONS 

In  summary,  we  used  a  case  study  approach  to  better  under¬ 
stand  how  civilian  health  plans  use  quality  and  value  measures 
to  monitor  their  performance  and  to  plan  and  evaluate  quality 
improvement  and  cost  reduction  efforts.  We  found  that  health 
plans  were  regularly  collecting  and  acting  on  quality  measures 
but  less  often  on  value  measures.  Some  measures,  particularly 
patient-reported  outcomes,  have  proven  to  be  difficult  to  collect 
and  use.  These  findings  may  be  useful  for  DoD  in  the  further 
development  of  its  own  psychological  health  care  quality  and 
value  measurement  programs. 
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